Skip to content

MariaDB Vector integrations for retriever & dataprep services #1645

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
chickenrae merged 5 commits into from
May 6, 2025
Merged

MariaDB Vector integrations for retriever & dataprep services #1645

chickenrae merged 5 commits into from
May 6, 2025

Conversation

@RazvanLiviuVarzaru
Copy link
Contributor

@RazvanLiviuVarzaru RazvanLiviuVarzaru commented May 5, 2025

Description

Add MariaDB Vector integrations for retriever & dataprep microservices.

Issues

n/a

Type of change

  • New feature (non-breaking change which adds new functionality)

Dependencies

  • retrievers Dockerfile: libmariadb-dev, build-essential
  • retrievers python requirments.txt: mariadb, langchain_mariadb
  • in dataprep Dockerfile: libmariadb-dev
  • dataprep python requirments.txt: mariadb, langchain_mariadb

Tests

The following tests will build the service docker image, run it and perform a series of tests against the exposed API endpoints.

cd tests
bash dataprep/test_dataprep_mariadb.sh
bash retrievers/test_retrievers_mariadb.sh

@RazvanLiviuVarzaru
Add MariaDB Vector third-party service
e3dc87e 
MariaDB Vector was introduced since MariaDB Server 11.7

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>
@RazvanLiviuVarzaru
Add retriever MariaDB Vector integration
05d41a7 
Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>
@RazvanLiviuVarzaru
Add dataprep MariaDB Vector integration
a0c00fd 
Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>
@RazvanLiviuVarzaru RazvanLiviuVarzaru mentioned this pull request May 5, 2025
Closed
1 task
@lvliang-intel
Copy link
Collaborator

lvliang-intel commented May 5, 2025

@RazvanLiviuVarzaru,
please fix the code scan issue.
Uploading image.png…

@RazvanLiviuVarzaru
Fix CI failures
535f4cd 
- md5 is used for the primary key not as a security hash
- fixed mariadb readme headers

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>
@RazvanLiviuVarzaru RazvanLiviuVarzaru force-pushed the feature/mariadb-vector branch from 42a3038 to 535f4cd Compare May 5, 2025 12:02
@RazvanLiviuVarzaru
Copy link
Contributor Author

RazvanLiviuVarzaru commented May 5, 2025

@lvliang-intel fixed in 535f4cd

thanks!

letonghan
letonghan approved these changes May 6, 2025
View reviewed changes
Copy link
Collaborator

@letonghan letonghan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @RazvanLiviuVarzaru for your contribution!

@chickenrae chickenrae merged commit 9cf40c1 into opea-project:main May 6, 2025
66 of 67 checks passed
@joshuayao joshuayao added this to the v1.4 milestone May 8, 2025
@joshuayao joshuayao added this to OPEA May 8, 2025
@joshuayao joshuayao moved this to Done in OPEA May 8, 2025
@joshuayao joshuayao added the feature New feature or request label May 8, 2025
jilongW pushed a commit to jilongW/GenAIComps that referenced this pull request May 12, 2025
@RazvanLiviuVarzaru @pre-commit-ci @jilongW
MariaDB Vector integrations for retriever & dataprep services (opea-p…
ba730fd 
…roject#1645)

* Add MariaDB Vector third-party service

MariaDB Vector was introduced since MariaDB Server 11.7

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

* Add retriever MariaDB Vector integration

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

* Add dataprep MariaDB Vector integration

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix CI failures

- md5 is used for the primary key not as a security hash
- fixed mariadb readme headers

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

---------

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: jilongw <jilong.wang@intel.com>
madison-evans pushed a commit to SAPD-Intel/GenAIComps that referenced this pull request May 12, 2025
@RazvanLiviuVarzaru @pre-commit-ci
MariaDB Vector integrations for retriever & dataprep services (opea-p…
77cc2b1 
…roject#1645)

* Add MariaDB Vector third-party service

MariaDB Vector was introduced since MariaDB Server 11.7

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

* Add retriever MariaDB Vector integration

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

* Add dataprep MariaDB Vector integration

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix CI failures

- md5 is used for the primary key not as a security hash
- fixed mariadb readme headers

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

---------

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
alexsin368 pushed a commit to alexsin368/GenAIComps that referenced this pull request May 15, 2025
@RazvanLiviuVarzaru @pre-commit-ci @alexsin368
MariaDB Vector integrations for retriever & dataprep services (opea-p…
11a79ff 
…roject#1645)

* Add MariaDB Vector third-party service

MariaDB Vector was introduced since MariaDB Server 11.7

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

* Add retriever MariaDB Vector integration

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

* Add dataprep MariaDB Vector integration

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix CI failures

- md5 is used for the primary key not as a security hash
- fixed mariadb readme headers

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

---------

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: alexsin368 <alex.sin@intel.com>
jilongW pushed a commit to jilongW/GenAIComps that referenced this pull request May 15, 2025
@RazvanLiviuVarzaru @pre-commit-ci @jilongW
MariaDB Vector integrations for retriever & dataprep services (opea-p…
ffeb881 
…roject#1645)

* Add MariaDB Vector third-party service

MariaDB Vector was introduced since MariaDB Server 11.7

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

* Add retriever MariaDB Vector integration

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

* Add dataprep MariaDB Vector integration

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix CI failures

- md5 is used for the primary key not as a security hash
- fixed mariadb readme headers

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

---------

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: jilongw <jilong.wang@intel.com>
yinghu5 added a commit that referenced this pull request May 16, 2025
@alexsin368 @pre-commit-ci
@yinghu5 @ZePan110 @lvliang-intel @MSCetin37 @RazvanLiviuVarzaru @chensuyue @Spycsh
Agent: add support for remote server (#1644)
fcf7a5a 
* add support for remote server

Signed-off-by: alexsin368 <alex.sin@intel.com>

* add steps to enable remote server

Signed-off-by: alexsin368 <alex.sin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove use_remote_service

Signed-off-by: alexsin368 <alex.sin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add OpenAI models instructions, fix format of commands

Signed-off-by: alexsin368 <alex.sin@intel.com>

* simplify ChatOpenAI instantiation

Signed-off-by: alexsin368 <alex.sin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Revert "simplify ChatOpenAI instantiation"

This reverts commit b7c4acf.

* add back check and logic for llm_engine, set openai_key argument

Signed-off-by: alexsin368 <alex.sin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Provide ARCH option for lvm-video-llama image build (#1630)

Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: alexsin368 <alex.sin@intel.com>

* Add sglang microservice for supporting llama4 model (#1640)

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>
Co-authored-by: Lv,Liang1 <liang1.lv@intel.com>
Signed-off-by: alexsin368 <alex.sin@intel.com>

* Remove invalid codeowner. (#1642)

Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: alexsin368 <alex.sin@intel.com>

* add support for remote server

Signed-off-by: alexsin368 <alex.sin@intel.com>

* add steps to enable remote server

Signed-off-by: alexsin368 <alex.sin@intel.com>

* remove use_remote_service

Signed-off-by: alexsin368 <alex.sin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: alexsin368 <alex.sin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: alexsin368 <alex.sin@intel.com>

* bug fix for chunk_size and overlap cause error in dataprep ingestion (#1643)

* bug fix for dataingest url

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* add validation function

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* validation update

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update validation function

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Mustafa <mustafa.cetin@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: alexsin368 <alex.sin@intel.com>

* MariaDB Vector integrations for retriever & dataprep services (#1645)

* Add MariaDB Vector third-party service

MariaDB Vector was introduced since MariaDB Server 11.7

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

* Add retriever MariaDB Vector integration

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

* Add dataprep MariaDB Vector integration

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix CI failures

- md5 is used for the primary key not as a security hash
- fixed mariadb readme headers

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

---------

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: alexsin368 <alex.sin@intel.com>

* update PR reviewers (#1651)

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: alexsin368 <alex.sin@intel.com>

* Expand test matrix, find all tests use 3rd party Dockerfiles (#1676)

* Expand test matrix, find all tests use 3rd party Dockerfiles

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: alexsin368 <alex.sin@intel.com>

* fix the typo of README.md Comp (#1679)

Update README.md for first entry of OPEA

Signed-off-by: alexsin368 <alex.sin@intel.com>

* Fix request handle timeout issue (#1687)

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: alexsin368 <alex.sin@intel.com>

* FEAT: Enable OPEA microservices to start as MCP servers (#1635)

Signed-off-by: alexsin368 <alex.sin@intel.com>

* Fix huggingface_hub API upgrade issue (#1691)

* Fix huggingfacehub API upgrade issue

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: alexsin368 <alex.sin@intel.com>

* add OpenAI models instructions, fix format of commands

Signed-off-by: alexsin368 <alex.sin@intel.com>

* Fix dataprep opensearch ingest issue (#1697)

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: alexsin368 <alex.sin@intel.com>

* Fix embedding issue with ArangoDB due to deprecated HuggingFace API (#1694)

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: alexsin368 <alex.sin@intel.com>

* simplify ChatOpenAI instantiation

Signed-off-by: alexsin368 <alex.sin@intel.com>

* Revert "simplify ChatOpenAI instantiation"

This reverts commit b7c4acf.

Signed-off-by: alexsin368 <alex.sin@intel.com>

* add back check and logic for llm_engine, set openai_key argument

Signed-off-by: alexsin368 <alex.sin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: alexsin368 <alex.sin@intel.com>
Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>
Signed-off-by: Mustafa <mustafa.cetin@intel.com>
Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>
Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Ying Hu <ying.hu@intel.com>
Co-authored-by: ZePan110 <ze.pan@intel.com>
Co-authored-by: Liang Lv <liang1.lv@intel.com>
Co-authored-by: Mustafa <109312699+MSCetin37@users.noreply.github.com>
Co-authored-by: Razvan Liviu Varzaru <45736827+RazvanLiviuVarzaru@users.noreply.github.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
Co-authored-by: Spycsh <39623753+Spycsh@users.noreply.github.com>
ZePan110 pushed a commit that referenced this pull request May 16, 2025
@jilongW @RazvanLiviuVarzaru
@pre-commit-ci @chensuyue @yinghu5
Added missing package for xtune running GPTQ model (#1684)
4a0976e 
* Update prepare_xtune.sh

Signed-off-by: jilongw <jilong.wang@intel.com>

* Update prepare_xtune.sh

Signed-off-by: jilongw <jilong.wang@intel.com>

* MariaDB Vector integrations for retriever & dataprep services (#1645)

* Add MariaDB Vector third-party service

MariaDB Vector was introduced since MariaDB Server 11.7

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

* Add retriever MariaDB Vector integration

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

* Add dataprep MariaDB Vector integration

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix CI failures

- md5 is used for the primary key not as a security hash
- fixed mariadb readme headers

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

---------

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: jilongw <jilong.wang@intel.com>

* update PR reviewers (#1651)

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: jilongw <jilong.wang@intel.com>

* Expand test matrix, find all tests use 3rd party Dockerfiles (#1676)

* Expand test matrix, find all tests use 3rd party Dockerfiles

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: jilongw <jilong.wang@intel.com>

* fix the typo of README.md Comp (#1679)

Update README.md for first entry of OPEA

Signed-off-by: jilongw <jilong.wang@intel.com>

* add version check

Signed-off-by: jilongw <jilong.wang@intel.com>

* add doc

Signed-off-by: jilongw <jilong.wang@intel.com>

* update doc

Signed-off-by: jilongw <jilong.wang@intel.com>

---------

Signed-off-by: jilongw <jilong.wang@intel.com>
Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>
Signed-off-by: chensuyue <suyue.chen@intel.com>
Co-authored-by: Razvan Liviu Varzaru <45736827+RazvanLiviuVarzaru@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
Co-authored-by: Ying Hu <ying.hu@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lvliang-intel lvliang-intel lvliang-intel approved these changes

@letonghan letonghan letonghan approved these changes

@lkk12014402 lkk12014402 Awaiting requested review from lkk12014402 lkk12014402 is a code owner

@XinyuYe-Intel XinyuYe-Intel Awaiting requested review from XinyuYe-Intel XinyuYe-Intel is a code owner

@ftian1 ftian1 Awaiting requested review from ftian1 ftian1 is a code owner

@chensuyue chensuyue Awaiting requested review from chensuyue chensuyue is a code owner

Assignees

No one assigned

Labels

feature New feature or request

Projects

OPEA
Status: Done

Milestone

v1.4

Development

Successfully merging this pull request may close these issues.

5 participants

@RazvanLiviuVarzaru @lvliang-intel @letonghan @joshuayao @chickenrae